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Scientific  and  Technicai  Objectives 

The  research  was  focused  on  the  development  of  a  synthetic  teammate  over  the  course  of 
a  year.  The  scientific  and  technical  objectives  were  to  develop  and  integrate 
computational  accounts  of  macrocognitive  processes  identified  as  necessary  for  working 
as  part  of  a  team  (e.g.,  language,  task  behavior,  situation  assessment/awareness,  etc.). 

Each  of  the  proposed  project  milestones  have  been  achieved;  1)  we  integrated  language 
comprehension,  agent-environment  interaction,  and  language  generation  components  into 
a  single  system  that  behaves  as  a  synthetic  teammate;  2)  we  developed  a  situation 
component  and  integrated  it  with  the  synthetic  teammate;  3)  we  continued  to  refine  the 
synthetic  teammate  through  test-develop-retest  iterations;  however,  instead  of  adding 
humans  as  teammates  we  developed  agents  that  were  low  in  cognitive  fidelity  using 
systems  of  finite  state  machines  that  acted  as  the  photographer  and  navigator.  The 
following  sections  provide  more  specifics  regarding  our  proposed  scientific  and  technical 
objectives. 

Background 

For  the  two  years  prior  to  the  current  ONR  award,  a  team  of  scientists  at  the  Cognitive 
Engineering  Research  Institute  (CERl,  an  independent  not-for-profit  research  institute  in 
Mesa,  AZ)  and  the  Performance  and  Learning  Models  team  (PALM)  at  the  Air  Force 
Research  Laboratory  (AFRL)  have  worked  on  the  development,  integration,  and 
validation  of  a  synthetic  teammate  (Myers  et  al.,  under  review).  The  synthetic  teammate 
will  interact  with  human  teammates  in  real-time  to  accomplish  a  reconnaissance  task 
within  an  Uninhabited  Air  Vehicle  synthetic  task  environment  (UAV-STE). 

The  UAV-STE  is  a  team-based  task  that  involves  three  interdependent  team 
members,  each  with  a  different  role.  The  team  members  are  the  Data  Exploitation 
Mission  Planning  and  Communications  operator  (DEMPC,  the  navigator)  who  is 
responsible  for  producing  a  dynamic  flight  plan,  including  speed  and  altitude  restrictions, 
an  Air  Vehicle  Operator  (AVO,  the  pilot)  who  controls  flight  systems,  and  a  Payload 
Operator  (PLO,  the  photographer)  who  monitors  sensor  equipment  and  photographs 
ground  targets.  The  team  members’  common  goal  is  to  photograph  ground  targets  and 
this  requires  interaction  between  team  members.  A  single  UAV-STE  mission  consists  of 
11-12  targets  and  lasts  a  maximum  of  40  minutes;  each  team  performs  five  40-minute 
missions. 

The  synthetic  teammate  will  act  as  the  AVO  in  the  UAV-STE,  and  is  being  developed 
using  the  ACT-R  computational  cognitive  architecture  (Anderson,  2007).  ACT-R  has 
been  under  continuous  development  for  several  decades  and  is  now  capable  of  accurately 
reproducing  human  microcognitive  processes  (e.g.,  memory  retrieval,  skill  acquisition, 
etc.)  Without  detailing  ACT-R,  cognition  revolves  around  the  interaction  between  a 
central  procedural  system  and  several  peripheral  modules.  There  are  modules  for  vision, 
motor  capabilities,  memory,  one  for  storing  the  model’s  intentions  for  completing  the 
task  (i.e.,  the  control  state),  and  a  module  for  storing  the  mental  representation  of  the  task 
at  hand  (problem  state,  see  Figure  1).  (For  more  detail  on  ACT-R,  see  Anderson,  2007.) 

We  have  chosen  ACT-R  for  two  important  reasons.  First,  there  is  an  abundance  of 
ACT-R  expertise  between  PALM  and  CERL  Second,  and  more  importantly,  ACT-R 
provides  a  good  foundation  to  investigate  how  macrocognitive  processes  (e.g.,  meta- 
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cognition,  situation  assessment/awareness,  etc.)  affect  microcognitive  processes,  and  vice 
versa  (Cooke  &  Myers,  2008).  Because  ACT-R  provides  good  quantitative  predictions  of 
human  performance  across  many  microcognitive  processes,  using  them  as  a  foundation 
for  developing  macrocognitive  processes  will  help  to  uncover  how  micro  and  macro 
processes  interact  within  complex  task  environments. 


Figure  I.  The  modules  of  the  ACT-R  6.0  computational  cognitive  architecture.  Adapted  from 

Anderson  (2007) 

Synthetic  teammate  development  has  been,  and  will  continue  to  be,  managed  through 
a  divide-and-conquer  strategy  across  a  set  of  components,  combined  with  a  synthesis 
strategy  for  component  integration.  To  support  synthesis  and  cognitive  plausibility  of  the 
three  major  components,  they  are  all  being  developed  using  the  ACT-R  architecture.  The 
major  components  include:  I)  language  comprehension,  2)  language  generation  and 
dialog  management,  and  3)  task  behavior.  Each  of  these  components  will  interact  through 
a  central  situation  component  (see  Figure  2). 


Figure  2.  Functional  components  of  the  synthetic  teammate 


Language  Comprehension 

The  language  comprehension  system  has  been  under  development  since  2002  (Ball, 
Heiberg  &  Silber,  2007).  It  is  based  on  a  linguistic  theory  of  the  grammatical  encoding  of 
referential  and  relational  meaning  (Ball,  2007a)  combined  with  a  theory  of  language 
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processing  based  on  the  activation,  selection,  and  integration  of  constructions 
corresponding  to  a  wide  range  of  linguistic  input  (Ball,  2007b).  The  component 
incrementally  processes  input  in  real-time,  constructing  a  linguistic  representation  that 
encodes  referential  and  relational  meaning. 

Language  Generation  and  Dialog  Modeling 

The  language  generation  and  dialog  component  was  developed  over  the  course  of  1 8 
months,  beginning  in  November  2006.  The  component  does  an  adequate  job  of  matching 
human  behavior  using  variabilized  “utterance  templates”  in  concert  with  a  strict 
constraint  hierarchy  based  on  principles  of  optimality  theory  (Prince  &  Smolensky, 
1993/2004).  The  constraint  hierarchy  is  implemented  using  ACT-R’s  spreading  activation 
mechanism.  Situational  constraints  activate  utterance  templates  that  are  instantiated  as 
chunks  within  ACT-R’s  declarative  memory,  and  the  template  with  the  greatest  activation 
is  selected  for  language  generation.  Despite  the  constraint  hierarchy,  noise  in  ACT-R’s 
spreading  activation  mechanism  allows  for  occasional  hierarchy  reversals.  Furthermore, 
the  constraint  hierarchy  is  adaptive,  resulting  in  variation  in  the  constraint  hierarchy  over 
time  as  a  function  of  communication  experience  with  different  teammates. 

Task  Behavior  Component 

The  task  behavior  component  has  been  under  development  since  summer  2007.  The  task 
model  was  developed  as  a  flat  goal-subgoal  hierarchy  based  on  task  environment 
constraints.  Task  goals  are  stored  as  chunks  in  declarative  memory,  and  are  retrieved 
when  appropriate.  For  example,  a  declarative  memory  chunk  representing  a  task-related 
activity  (i.e,  a  goal)  is  retrieved  from  memory  as  a  function  of  the  model’s  situation.  As 
the  situation  changes,  different  goal  chunks  are  retrieved  from  memory,  producing 
flexible  and  robust  goal  selection  and  execution. 

Situation  Component 

The  situation  component  was  implemented  as  one  ACT-R  declarative  memory  chunk  that 
was  stored  in  the  synthetic  teammate’s  problem  state  buffer,  and  was  used  in  a  similar 
capacity  to  the  “blackboard”  in  blackboard  architectures  used  in  artificial  intelligence 
research.  The  chunk  contained  49  slots  for  storing  information  associated  with  task 
environment  and  language  generation  states.  As  the  synthetic  teammate  operated  as  the 
AVO  in  the  UAV  reconnaissance  task,  information  within  the  situation  component  chunk 
was  updated  to  reflect  the  state  of  the  task  environment  and  communications  between 
teammates.  The  information  held  in  the  situation  component  is  used  to  retrieve  and 
execute  specific  goals,  leading  to  situated  cognitive  control  within  each  functional 
component  of  the  synthetic  teammate  (see  Figure  2).  Information  held  in  the  situation 
component  implementation  did  not  decay,  making  situation  information  potentially 
eternally  available  to  the  synthetic  teammate,  going  well  above  human  situation 
assessment/awareness  capabilities. 

Integrated  Components 

The  task  behavior  and  the  language  generation  and  dialog  components  were  combined 
into  a  single,  integrated  cognitive  system  (Gray,  2007)  capable  of  behaving  in  the  UAV- 
STE  and  sharing  task-related  information  with  teammates  as  a  function  of  a 


representation  stored  in  the  situation  component  prior  to  the  current  ONR  award.  A  key 
to  this  initial  integration  was  the  introduction  of  implicit  task  switching  between  the  two 
components. 
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Background  Summary 

There  have  been  significant  advances  made  in  developing  a  cognitively  plausible  model 
that  that  can  interact  with  teammates  using  the  ACT-R  architecture.  First,  language 
comprehension  capabilities  have  been  steadily  advancing  over  the  past  two  years. 

Second,  language  generation  capabilities  have  advanced  to  the  point  that  the  synthetic 
teammate  is  capable  of  generating  utterances  based  on  a  primitive  situation 
representation.  Third,  knowledge  and  rules  enabling  task-relevant  behavior  have  been 
integrated  with  the  language  generation  component,  producing  a  composite  component. 

Although  much  progress  was  made,  more  needed  to  occur  to  ultimately  reach  the  goal 
of  developing  a  synthetic  teammate  that  can  interact  with  humans  in  real-time  and 
provide  insight  into  how  macrocognitive  and  microcognitive  process  interact.  The 
following  section  covers  the  statement  of  work  for  the  current  ONR  award. 

Statement  of  Work 

The  following  sections  detail  the  planned  improvements  to  the  situation  component, 
deliverables,  and  milestones  associated  with  the  current  ONR  award. 

Planned  Improvements  to  the  Situation  Component 

The  situation  component  is  important  for  representing  complex  agent  states  resulting 
from  agent— task-team  interactions.  Inputs  to  the  situation  component  will  include  task 
environment  states  (e.g.,  the  UAV’s  current  altitude,  speed,  course,  etc.)  combined  with 
communication  from  teammates  (e.g.,  desired  UAV  airspeeds  and  altitudes,  upcoming 
targets,  etc.),  and  background  knowledge  (e.g.  information  about  the  other  teammates). 
Development  of  the  situation  component  will  contribute  some  of  the  groundwork 
necessary  for  developing  high-level,  or  macrocognitive,  processes  (e.g.,  reasoning  over 
complex  scenarios,  simulating  others  beliefs,  predicting  future  states,  etc.)  within  a 
computational  cognitive  architecture  that  operates  at  a  relatively  low-level  of  cognitive 
and  perceptual/motor  behavior  (i.e.,  ACT-R  'operates'  at  50  ms  cycles,  Anderson,  2007). 

Although  the  blackboard  implementation  has  worked  well  for  developing  and 
integrating  the  task  behavior  and  language  generation  and  dialog  components,  the 
situation  component  must  be  improved  to  reflect  human  capabilities.  Furthermore, 
improvement  to  the  situation  component  will  facilitate  the  integration  of  the  language 
comprehension  component  and  provide  a  sufficient  foundation  for  other  macrocognitive 
processes  (i.e,  reasoning,  predicting  future  states,  etc.)  within  the  ACT-R  architecture. 

ITie  situation  component  will  combine  relevant  information  gleaned  from  linguistic 
and  task  environment  inputs,  supporting  language  generation  and  continuing  task 
behavior.  Different  from  the  blackboard  implementation,  the  planned  implementation  of 
the  situation  component  will  be  primarily  propositional  in  nature — information  and 
relationships  between  information  described  in  the  linguistic  input  and  derived  from  the 
task  environment  and  background  knowledge  will  be  encoded  in  a  propositional  manner. 
Because  the  synthetic  teammate  is  modeled  using  ACT-R,  propositional  representations 
will  ultimately  be  implemented  using  ACT-R’s  chunk-based  representations. 
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Integrating  the  Languaee  Comprehension  Component.  A  key  commitment  of  the 
language  comprehension  component  is  a  general  capability  to  handle  basic  grammatical 
constructions  of  English,  enabling  use  of  the  component  across  different  applications  and 
models.  Issues  like  lexical  and  structural  ambiguity,  ungrammatical  inputs,  and 
variability  in  input  forms  cannot  be  ignored.  Despite  the  advanced  state  of  the 
component,  the  requirements  for  adequate  language  comprehension  over  an  open-ended 
range  of  input  are  significant,  and  much  remains  to  be  done. 

Another  key  commitment  is  the  position  that  adherence  to  well-established 
constraints  on  human  language  processing  may  actually  facilitate  the  goal  of  building  a 
functional  language  comprehension  component  (Ball,  2006).  For  example,  the  common 
use  of  a  separate  part-of-speech  tagging  routine  whose  output  feeds  a  separate  parsing 
process  is  eschewed  in  favor  of  an  integrated  approach  that  is  capable  of  operating 
incrementally  in  real-time. 

Moving  the  situation  component  to  a  propositional  format  will  facilitate  integrating 
the  language  comprehension  component  with  the  task  behavior/language  generation 
system  since  the  linguistic  representations  derived  from  the  language  comprehension 
component  contain  much  of  the  information  needed  to  generate  propositional 
representations.  Complete  component  integration  will  result  in  a  combined  computational 
cognitive  model  that  is  likely  to  exceed  most  existing  cognitive  models  in  size.  Questions 
of  efficiency  and  complexity  will  need  to  be  addressed  to  ensure  that  the  synthetic 
teammate  is  capable  of  functioning  in  real-time  without  exceeding  resource  capacities 
within  the  timeframe  for  a  mission  in  the  UAV-STE  (~40  minutes). 

Continued  Development  with  Humans  In-the-Loop.  The  development  of  the  synthetic 
teammate  will  be  in  vain  without  validation,  and  we  view  validation  as  more  than 
producing  a  working  agent.  It  is  possible  that  the  synthetic  teammate’s  interactions  are 
unnatural,  disruptive,  or  unfaithful  to  those  of  humans  performing  the  same  task  even 
though  it  adequately  pilots  the  UAV  in  the  STE.  Consequently,  we  plan  on  beginning 
human-in-the-loop  validation  procedures  as  soon  as  practically  possible.  Looking  to 
previous  research  on  synthetic  teammates  for  direction,  it  is  clear  that  validation  is  rarely 
accomplished,  or  goes  beyond  anecdotal  comments  from  human  participants.  We  are 
planning  a  hierarchical  validation  procedure: 

•  Key-press  level  —  comparing  model  data  to  human  data  within  a  goal,  such  as 
setting  airspeed,  altitude,  course,  current  waypoint,  sending  a  message,  etc. 

•  Goal-selection  level  —  comparing  goal  execution  sequences  to  humans  (general 
order  of  goals  executed  to  reach  waypoints.  This  level  of  analysis  will  help 
validate  the  situation  component,  as  the  component  has  a  direct  influence  on 
which  goal  is  selected  and  when. 

•  Mission  level  -  comparing  synthetic  teammate  mission  performance  to  human 
performance  across  a  set  of  UAV-STE  missions. 

•  Team  level  —  comparing  team  performance  between  teams  that  have  an 
incorporated  synthetic  teammate  and  all-human  teams. 

Validation  at  the  key-press  and  goal-selection  levels  began  with  official  start  of  the 
proposed  research  (Myers,  2009).  Mission  level  and  team  level  validation  procedures  will 
begin  on  once  the  synthetic  teammate  is  fully  capable  of  interacting  with  human 
teammates. 
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Deliverables 

Prototype  demonstration*  8/01/2009 

Final  technical  report  1/31/2010 

Publication  and  presentation  1/31/2010 

*  We  recognized  that  a  fully  functional  and  valid  synthetic  teammate  was  likely 
unattainable  within  a  year;  however  we  nonetheless  planned  to  have  a  partially  functional 
prototype  to  showcase  the  accomplishments  of  our  year-long  effort.  Also  in  the  interest 
of  risk  mitigation  this  effort  has  advanced  the  state-of-the-art  of  cognitive  modeling  and 
synthetic  teammates  in  the  following  ways  (Ball,  Myers,  Heiberg,  et  al.,  submitted): 

■  Improved  understanding  for  managing  and  coordinating  the  development  of  large 
scale  computational  cognitive  models 

•  Improved  understanding  of  how  to  develop  macrocognitive  processes  in  a 
cognitive  architecture  situated  around  microcognitive  processes. 

■  Advantages  and  disadvantages  of  propositional  representations  within  the 
situation  component 

■  Advantages  and  disadvantages  of  using  a  situation  component  as  a  large  scale 
model’s  form  of  cognitive  control 

■  Advantages  and  disadvantages  associated  with  modeling  approaches  for  each  of 
the  components 

■  Improved  understanding  about  solutions  to  software  and  hardware  issues 
associated  with  developing  large  scale  models 

Project  Schedule  and  Milestones 


Date:  Milestone: 


December  1,  2008 

Project  kick-off 

March  1,2009 

Situation  component  specification  finalized 

June  1,  2009 

Begin  integrating  language  comprehension  component  with  task 
behavior/language  generation  and  dialog  management  system. 

August  1,  2009 

Prototype  demonstration 

September  1,  2009 

Begin  model  validation  with  teammates 

November  30,  2009 

Project  end 

Summary  of  Scientific  and  Technical  Objectives,  Background,  & 
Approach 

The  synthetic  teammate  is  being  developed  using  the  ACT-R  computational  cognitive 
architecture  (Anderson,  2007).  The  synthetic  teammate  will  interact  with  two  humans  in 
real-time  to  accomplish  a  UAV  reconnaissance  task  within  a  synthetic  task  environment. 
The  UAV-STE  is  a  dynamic  command  and  control  task  which  requires  three  teammates 
(photographer,  navigator,  and  pilot)  to  interact  with  each  other  to  attain  the  common  goal 
of  photographing  as  many  reconnaissance  targets  as  possible  over  a  the  course  of 
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multiple  40-minute  missions.  The  synthetic  teammate  plays  the  role  of  the  pilot  in  the 
UAV-STE. 

Four  macrocognitive  components  have  been  identified  for  the  synthetic  teammate  to 
adequately  interact  with  its  environment  and  teammates:  language  comprehension, 
language  generation/dialog  management,  situation  assessment,  and  agent-environment 
interaction.  As  part  of  a  larger  collaboration  with  scientists  at  AFRL  and  Michael 
Matessa  of  Alion,  each  of  the  components  have  been  developed  and  integrated,  facilitated 
through  the  use  of  the  ACT-R  cognitive  architecture.  The  language  generation/dialog 
management  component  was  successfully  integrated  with  the  agent-environment 
interaction  component  prior  to  the  beginning  of  the  current  ONR  award.  The  current 
ONR  award  provided  support  for  the  successful  development  and  integration  of  the 
situation  assessment  and  language  comprehension  components  with  the  agent- 
environment  interaction  and  language  generation  components. 

The  refinement  of  each  synthetic  teammate  macrocognitive  component,  as  well  as  the 
interactions  among  them,  have  benefited  from  development  iteration  with  teammates  in 
the  loop.  Failures  of  the  model  working  with  teammates  have  provided  fodder  for  further 
development  of  the  synthetic  teammate’s  components.  Eventually,  the  synthetic 
teammate  will  provide  insight  into  how  individual  teammates  affect  team  processes,  and 
how  team  processes  affect  individuals’  cognitive  processes. 

Concise  Accomplishments 

(Accomplishments  resulting  directly  from  the  current  ONR  award  are  italicized) 

•  Major  expansion  of  the  linguistic  coverage  of  the  language  comprehension 
component  (language  comprehension:  Dr.  Jerry  Ball  of  AFRL) 

•  Addition  of  an  externalized,  persistent  Declarative  Memory  (DM)  system  to 
ACT-R  6  (persistent  DM:  Drs.  Scott  Douglass  and  Jerry  Ball  of  AFRL) 

•  Significant  improvement  in  the  handling  of  variability  in  the  linguistic  input 
(linguistic  input  variability:  Mrs.  Mary  Freiman  of  L3  Communications) 

•  Modification  of  the  ACT-R  cognitive  architecture  to  improve  the  word 
recognition  subcomponent  (improved  word  recognition:  Mary  Freiman  of  L3 
Communications) 

•  Situation  assessment  component  specified  in  sufficient  detail  to  support 
processing  of  scripted  communication  and  is  poised for  expansion  beyond 
scripted  communication  (situation  assessment) 

•  Situation  assessment  component  fully  integrated  with  synthetic  teammate  model 
(full  integration) 

•  Specification  (Dr.  Scott  Douglass)  and  development  of  low  cognitive  fidelity 
teammates  in-the-loop  development  and  testing  of  synthetic  teammate  (low- 
fidelitv  agents) 

•  Key  press  level  model  validation  effort  demonstrated  an  excellent  fit  between 
model  and  human  data  (kev-press  level  validation) 

•  Demonstration  of  synthetic  teammate  after  integration  of  language 
comprehension,  agent-environment  interaction,  language  generation,  and 
situation  components  (demonstration) 
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Data  analysis  of  human  teams  from  an  experiment  that  manipulated 
communication  mode  (text-chat  vs,  audio)  that  serves  as  baseline  data  for 
comparisons  with  the  synthetic  teammate  incorporated  in  teams  (further  analyses) 


Expanded  Accomplishments 

Language  comprehension.  A  key  commitment  of  the  project  was  for  the  synthetic 
teammate  to  be  capable  of  handling  a  broad  range  of  linguistic  inputs.  To  support  this,  the 
language  comprehension  component  has  been  significantly  expanded  over  the  last  year. 

In  particular,  the  language  comprehension  component  is  now  capable  of  handling  a  broad 
range  of  linguistic  constructions  including  declarative,  interrogative  and  imperative 
sentences  of  various  types,  sentences  including  a  broad  range  of  verbal  constructions 
including  intransitive,  transitive,  and  ditransitive  verbs,  and  verbs  which  take  a  clausal  or 
locative  complement.  Besides  expanding  the  general  linguistic  coverage,  the  language 
comprehension  component  has  been  expanded  to  handle  the  specific  constructions  that 
occur  in  the  corpus  collected  in  past  experiments.  These  include  existential  there 
constructions  (e.g.  ''there  are  no  restrictions”),  predicate  nominal  constructions  (e.g.  "the 
next  waypoint  is  a  target”),  and  contractions  of  subject  and  auxiliary  (e.g.  "It’s  a  target”). 
Efforts  are  currently  underway  to  develop  mechanisms  for  automatically  expanding  the 
lexical  coverage  of  the  language  comprehension  component.  In  particular,  we  are  using 
the  British  National  Corpus  and  WordNet  as  resources  for  expanding  the  coverage  of  the 
system.  To  take  advantage  of  these  resources,  the  lexical  entries  they  contain  must  be 
mapped  into  the  lexical  ontology  used  by  the  language  comprehension  component.  We 
are  very  near  to  having  a  capability  to  do  this. 

Persistent  DM.  To  meet  the  large  declarative  memory  (DM)  requirements  of  the  synthetic 
teammate,  we  have  developed  an  external  DM  storage  and  retrieval  capability  in  ACT-R 
6  based  on  PostgreSQL,  a  powerful,  open  source  object-relational  database  management 
system  (DBMS).  This  "persistent-DM”  module  outsources  the  storage  of  ACT-R  DM 
elements,  or  chunks,,  to  an  industrial-strength  external  DBMS  while  leaving  ACT-R’s 
DM  retrieval  calculus  untouched.  The  persistent-DM  module  modifies  the  behavior  of 
ACT-R’s  declarative  module  by:  (1)  introducing  seven  control  parameters;  (2)  providing 
programmatic  support  for  managing  the  interaction  between  ACT-R  and  the  PostgreSQL 
DBMS;  (3)  extending  the  retrieval  process;  and  (4)  modifying  the  comparison  of  chunk 
slots.  The  addition  of  the  persistent-DM  DBMS  does  not  interfere  with  ACT-R 
predictions  associated  with  the  retrieval  and  storage  of  declarative  knowledge,  but  instead 
outsources  the  declarative  system  from  Lisp  to  improve  computational  efficiency. 

Linguistic  input  variability  and  improved  word  recognition.  Whereas  previous 
experiments  were  based  on  audio  communications,  the  most  recent  experiment  (funded 
by  AFOSR  in  the  precursor  to  the  current  ONR  award)  involved  text  messaging  and  we 
now  have  a  corpus  of  text  messages  that  reveals  tremendous  variability  in  the  form  of  the 
linguistic  input  (i.e.,  misspellings,  abbreviations,  etc.).  The  word  recognition 
subcomponent  of  the  language  comprehension  component  has  been  significantly 
modified  to  handle  this  variability.  In  addition,  the  perceptual  encoding  mechanism  of  the 
ACT-R  cognitive  architecture  was  modified  to  support  the  perception  of  a  perceptual 


span  that  is  not  limited  to  space  delimited  words  as  in  ACT-R’s  default  mechanism.  This 
modification  improves  the  performance  of  the  word  recognition  component  at  the  same 
time  that  it  improves  cognitive  fidelity. 

Situation  assessment.  The  situation  assessment  component  of  the  synthetic  teammate 
represents  the  current  situation  as  informed  by  the  linguistic  input,  the  task  environment, 
the  discourse  context,  and  background  knowledge.  Further,  the  situation  component 
constitutes  the  primary  meaning  representation  of  the  system,  although  the  linguistic 
representations  that  are  mapped  into  the  situation  component  also  encode  important 
aspects  of  meaning.  Consequently,  the  situation  component  grounds  the  meaning  of 
referring  expressions  in  the  linguistic  input  in  the  objects  and  situations  from  the  task 
environment,  discourse  context  and  background  knowledge,  all  of  which  can  be  stored  in 
the  situation  component. 

We  have  developed  a  situation  component  that  handles  propositional  and  discourse 
content,  and  designed  the  component  to  eventually  handle  spatial  and  imaginal  content 
within  the  ACT-R  cognitive  architecture.  The  result  was  a  new  ACT-R  module  that 
contains  eight  buffers.  The  buffers  are  used  for  maintaining  declarative  memory  chunks 
that  represent  the  perceived  situation  of  the  synthetic  teammate.  These  chunks  are 
retrieved  from  memory  and  used  to  process  received  communications,  reason,  and  make 
decisions. 

Full  integration.  The  initial  integration  of  the  four  cognitive  components  of  the  synthetic 
teammate  was  grounded  in  the  ability  of  the  integrated  synthetic  teammate  to  handle  the 
communications  required  to  fly  thru  the  first  two  waypoints  of  a  UAV  reconnaissance 
mission.  Because  the  amount  of  communications  required  to  achieve  this  is  limited,  we 
refer  to  these  as  scripted  communication.  For  the  initial  integration,  the  focus  was  on 
getting  the  components  to  work  together.  Once  this  was  achieved,  the  focus  shifted  to 
expanding  the  capability  of  the  synthetic  teammate  to  process  a  broader  range  of 
communications  and  behaviors  over  the  course  of  a  40-minute  reconnaissance  mission 
involving  more  than  10  waypoints.  The  language  comprehension  component  is  currently 
capable  of  generating  the  situation  representations  corresponding  to  communications 
from  the  scripted  communications.  The  agent-environment  interaction  component  is 
capable  of  responding  to  changes  in  the  situation  assessment  component,  such  as 
changing  airspeed,  altitude,  waypoints,  et  cetera.  The  language  generation/dialog 
management  component  is  capable  of  generating  the  synthetic  teammate’s  text-based 
communications  for  responding  to  comprehended  communications  and/or  providing 
updates  to  teammates  on  the  performance  of  the  UAV. 

Low-fidelity  agents.  In  order  to  test  the  synthetic  teammate’s  macrocognitive  processes 
when  interacting  with  teammates,  we  developed  low-cognitive-fidelity  photographer  and 
navigator  agents  using  systems  of  finite  state  automata.  The  formalism  for  developing  the 
agents  was  developed  by  Dr.  Scott  Douglass  at  AFRL  as  an  existence  proof  for 
developing  models  using  a  hybrid  of  text  and  a  graphical  modeling  language  (e.g.. 
General  Modeling  Environment,  GME)  that  can  be  automatically  transformed  into  an 
executable  model.  The  models  of  the  photographer  and  navigator  were  developed  by  the 
PI  of  the  grant  and  interact  with  the  synthetic  teammate  through  the  instant  messaging 
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system.  The  models  can  be  scaled-up  in  a  simple  manner  to  produce  communications 
based  on  increased  environmental  complexity  and  were  developed  to  eliminate  the  need 
and  cost  of  having  humans  play  the  role  of  the  synthetic  teammate’s  navigator  and 
photographer. 

Key-press  level  validation.  An  effort  to  validate  the  agent-environment  interaction 
component  of  the  synthetic  teammate  at  the  key  press  and  strategy  level  of  analysis  was 
completed.  The  results  demonstrated  that  the  model  adequately  approximated  the 
performance  of  human  operators  performing  for  necessary  setting  tasks  for  flying  the 
UAV  from  one  location  to  another:  setting  the  airspeed,  course,  altitude,  and  new 
location.  Consequently,  the  model  is  a  good  approximation  to  human  performance  at  the 
key  press  and  setting  task  strategy  levels  of  analysis.  The  next  validation  effort  of  the 
agent-environment  interaction  component  is  to  compare  the  synthetic  teammate’s 
strategies  for  choosing  which  setting  task  is  performed,  and  the  environment  state/context 
when  it  was  selected  (Myers,  2009). 

Demonstration.  A  demonstration  was  performed  for  Dr.  Paul  Bello,  the  project’s  manager 
at  the  Office  of  Naval  Research.  A  video  version  of  the  demonstration  provided  to  Dr. 
Bello  is  included  as  supplemental  information  with  this  report. 

Further  analyses.  Recent  data  analyses  have  revealed  that  there  is  no  statistical  difference 
in  team  performance  between  teams  that  communicate  using  audio  and  teams  that 
communicate  with  a  text-based  communication  system.  However,  evidence  has  emerged 
that  teams  using  text-based  communications  coordinate  in  patterns  that  differ  from 
communication  patterns  of  teams  that  use  audio  communications.  Specifically,  there  is  an 
inherent  lag  in  communication  reception  when  using  text-based  communications  that 
does  not  occur  in  audio-based  communications.  The  average  lag  time  for  the  text-based 
communications  was  10.52  seconds.  A  3  (teammate)  x  4  (mission)  mixed  Analysis  of 
Variance  (ANOVA)  revealed  that  communication  reception  lag  was  a  function  of  mission 
and  teammate,  where  the  lag  decreased  as  mission  number  increased,  but  did  so 
differentially  based  on  teammate  role  (navigator,  pilot,  photographer,  see  Figure  1). 
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Figure  1.  The  teammate  x  mission  interaction  effect  on  communication  lag  time. 

Team  coordination  (K)  was  computed  using  a  ratio  of  times  associated  with  key 
components  of  information  sharing,  such  as  information  associated  with  upcoming 
targets  (i.e.,  information;  /»),  altitude  and  airspeed  negotiation  between  the  photographer 
and  the  pilot  (negotiation;  N^),  and  feedback  from  the  photographer  that  a  good  picture 
was  taken  (feedback;  F„): 


The  formula  for  calculating  coordination  score  (K)  has  been  used  in  previous  experiments 
to  determine  differences  in  coordination  dynamics  between  treatment  groups  in  team 
experiments  (Cooke,  Gorman,  Pedersen,  et  al.,  2007).  To  determine  if  there  was  a 
difference  in  coordination  score  between  audio-comm  and  text-comm  teams,  a  2 
(communication  mode)  x  4  (mission)  mixed  ANOVA  was  conducted  on  coordination 
scores.  There  was  a  significant  main  effect  for  which  text-comm  had  a  significantly  lower 
coordination  score  than  audio-comm  (p  =  0.042).  This  is  not  to  say  that  the  audio  comm 
condition  coordinated  "better"  but  only  to  say  that  the  two  communication  conditions 
coordinated  differently.  Further,  a  measure  that  reveals  the  stability  of  team  coordination 
dynamics,  the  Hurst  exponent,  was  also  analyzed  to  determine  if  there  was  a  coordination 
stability  difference  between  communication  groups.  An  independent  samples  /-test  on  the 
average  Hurst  exponents  across  teams  revealed  that  text-comm  teams  were,  on  average, 
coordinated  in  a  more  stable  fashion  (A/=  0.9527,  SD  =  0.0131)  than  voice-comm  teams 
(M=  0.8988,  SD  =  0.061),  /(1 5)  =  2.287, p  =  0.037. 
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Major  Problems/lssues 

There  are  no  major  problems  other  than  the  sheer  amount  of  work  necessary  to  get  to  the 
point  of  bringing  human  teammates  into  the  development  loop.  Mentioned  above,  the 
integration  of  the  situation  assessment  and  language  comprehension  components  with  the 
rest  of  the  synthetic  teammate  has  proven  to  be  a  challenge.  The  challenge  results  from 
making  changes  to  the  components  in  order  to  accommodate  integration  while  at  the 
same  time  maintaining  the  efficacy  of  each  individual  component.  Although  this  is  not  a 
“major  problem”  it  should  be  noted  that  integration  is  also  not  trivial. 

A  further  issue  is  synthetic  teammate/CERTT  UAV-STE  integration.  This  type  of 
integration  issue  is  a  regular  occurrence  when  integrating  any  modeling  formalism  with  a 
dynamic  synthetic  task  environment.  Stuart  Rodgers  and  Dr.  Steven  Shope  have 
developed  a  mechanism  for  providing  environment  information  to  the  synthetic 
teammate,  which  is  currently  under  refinement  to  improve  system  response  times. 

Technology  Transfer 

•  Interactions  with  Dr.  Greg  Traflon  and  Dr.  Raj  Ratwani  of  the  Naval  Research 
Laboratory  regarding  approaches  to  embodied  models  of  situation 
awareness/assessment. 

•  Interactions  with  Dr.  Wink  Bennett,  Craig  Eidman  and  Dr.  Gary  Boyle  at  the  Air 
Force  Research  Laboratory  on  training  applications  of  synthetic  wingmen  in  the 
Joint  Terminal  Attack  Controller  Dome  at  the  Air  Force  Research  Laboratory  in 
Mesa,  AZ. 

•  Interactions  with  the  US  Air  Force  Scientific  Advisory  Board  to  provide  input  to 
their  study  on  the  application  of  cognitive  modeling  to  Virtual  Training 
Technologies. 
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